Organic learning

ABSTRACT

Certain aspects of the present disclosure provide systems and methods for configuring and training neural networks. The method includes models of individual neurons in a network that avoid certain biologically impossible or implausible features of conventional artificial neural networks. Exemplary networks may use patterns of local connections between excitatory and inhibitory neurons to provide desirable computational properties. A network configured in this manner is shown to solve a digit classification problem.

BACKGROUND Field

Certain aspects of the present disclosure generally relate to neuralsystem engineering, and more particularly to systems and methods forconfiguring and/or training neural networks for classification.

Background

The last several years have seen significant advances in the applicationof artificial neural networks to machine learning problems. Examplesinclude the application of neural networks to visual classificationtasks, auditory classification tasks, and the like, for which artificialneural networks have achieved state-of-the-art performance. In the viewof many neuroscientists, however, this progress has not translated intoincreased understanding of biological intelligence. In addition,principles of biological neural networks have not informed the design ofartificial neural networks in many respects.

To the extent that conventional artificial neural networks ignore oreven contradict certain principles of biological neural networkstructure and function, progress towards achieving certain aspects ofbiological intelligence may be hampered. Accordingly, certain aspects ofthe present disclosure are directed to configuring and training neuralnetworks that may be reconciled with the structure and function ofbiological neural networks.

SUMMARY

Certain aspects of the present disclosure generally relate to providing,implementing, and using a method of configuring neural networks.According to certain aspects, a visual data classification network maybe configured such that much of the training typically associated withneural network design may be avoided.

Certain aspects of the present disclosure provide a system forconfiguring a neural network. The system generally includes a memory anda processor coupled to the memory. The processor is configured to:arrange a first plurality of neurons in a topography, wherein aselectivity of a first neuron of the first plurality is based at leastin part on a position of the first neuron in the topography; arrange asecond plurality of neurons in the topography; wherein a second neuronof the plurality of neurons includes a body and a dendrite, the bodyhaving a position in the topography and the dendrite extending from thebody and having an orientation in the topography. The processor isfurther configured to: connect the first neuron to the second neuron, sothat an output of the first neuron is an input to the second neuron,based at least in part on: the position of the first neuron; theorientation of the dendrite at the position of the first neuron; and theselectivity of the first neuron.

Certain aspects of the present disclosure provide a non-transitorycomputer readable medium having instructions stored thereon. Theinstructions, upon execution by a computing device, cause the computingdevice to perform operations comprising: arranging a first plurality ofneurons in a topography, wherein a selectivity of a first neuron of thefirst plurality is based at least in part on a position of the firstneuron in the topography; arranging a second plurality of neurons in thetopography; wherein a second neuron of the plurality of neurons includesa body and a dendrite, the body having a position in the topography andthe dendrite extending from the body and having an orientation in thetopography. The instructions, upon execution by a computing device,cause the computing device to perform operations further comprising:connecting the first neuron to the second neuron, so that an output ofthe first neuron is an input to the second neuron, based at least inpart on: the position of the first neuron; the orientation of thedendrite at the position of the first neuron; and the selectivity of thefirst neuron.

Certain aspects of the present disclosure provide a method forconfiguring a neural network. The method generally includes: arranging afirst plurality of neurons in a topography, wherein a selectivity of afirst neuron of the first plurality is based at least in part on aposition of the first neuron in the topography; arranging a secondplurality of neurons in the topography; wherein a second neuron of theplurality of neurons includes a body and a dendrite, the body having aposition in the topography and the dendrite extending from the body andhaving an orientation in the topography. The method further includesconnecting the first neuron to the second neuron, so that an output ofthe first neuron is an input to the second neuron, based at least inpart on: the position of the first neuron; the orientation of thedendrite at the position of the first neuron; and the selectivity of thefirst neuron.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates example digits with added Poisson noise in accordancewith certain aspects of the present disclosure.

FIG. 2A illustrates post-synaptic potentials using a step function inaccordance with certain aspects of the present disclosure.

FIG. 2B illustrates post-synaptic potentials using a rise and decay timein accordance with certain aspects of the present disclosure.

FIG. 3 illustrates a network summary in accordance with certain aspectsof the present disclosure.

FIG. 4A illustrates a connection arrangement to form a one-dimensionaledge detector in accordance with certain aspects of the presentdisclosure.

FIG. 4B illustrates a connection arrangement to form a one-dimensionalline detector in accordance with certain aspects of the presentdisclosure.

FIG. 5 illustrates weight templates in accordance with certain aspectsof the present disclosure.

FIG. 6A illustrates a neuron with star-shaped dendrites havingconnections to line-detector neurons that are responsive to orientationssubstantially perpendicular to the dendrite orientation in accordancewith certain aspects of the present disclosure.

FIG. 6B illustrates a neuron that is responsive to the presence of acircular path in its receptive field in accordance with certain aspectsof the present disclosure.

FIG. 6C illustrates a receptive field of a neuron in accordance withcertain aspects of the present disclosure.

FIG. 6D illustrates a neuron that is responsive to the presence of asemi-circular path in its receptive field in accordance with certainaspects of the present disclosure.

FIG. 6E illustrates a population of neurons that are responsive to thepresence of a semi-circular path in accordance with certain aspects ofthe present disclosure

FIG. 7A illustrates a neuron having connections to line-detector neuronsthat are responsive to orientations substantially parallel to thedendrite orientation in accordance with certain aspects of the presentdisclosure.

FIG. 7B illustrates a neuron that is responsive to the presence of acorner in its receptive field, with the locus at the position of theneuron body in accordance with certain aspects of the presentdisclosure.

FIG. 7C illustrates a receptive field of a neuron in accordance withcertain aspects of the present disclosure.

FIG. 8A illustrates a neuron with star-shaped dendrites havingconnections to edge detector neurons that are responsive to orientededges substantially perpendicular to the dendrite orientation inaccordance with certain aspects of the present disclosure.

FIG. 8B illustrates a neuron that is responsive to the presence of anenclosed circle in its receptive field in accordance with certainaspects of the present disclosure.

FIG. 9A illustrates a neuron having connections to line-detector neuronsin accordance with certain aspects of the present disclosure.

FIG. 9B illustrates a neuron that is responsive to texture features inaccordance with certain aspects of the present disclosure.

FIG. 10A illustrates neurons arranged in a topography for auditoryprocessing in accordance with certain aspects of the present disclosure.

FIG. 10B illustrates example auditory receptive fields in accordancewith certain aspects of the present disclosure.

FIG. 10C illustrates a neuron having connections to auditory edgedetector neurons in accordance with certain aspects of the presentdisclosure.

FIG. 10D illustrates a neuron that is responsive to an auditory patternin accordance with certain aspects of the present disclosure.

FIG. 11 illustrates a summary of neurons responsivity types in a neuralnetwork in accordance with certain aspects of the present disclosure.

FIG. 12 illustrates a number of neurons by position in a layer inaccordance with certain aspects of the present disclosure.

FIG. 13 illustrates non-digit distractors in accordance with certainaspects of the present disclosure.

FIG. 14 illustrates example results for an untrained neural network inaccordance with certain aspects of the present disclosure.

FIG. 15 illustrates example accuracy results as a neural network istrained in accordance with certain aspects of the present disclosure

DETAILED DESCRIPTION

The detailed description set forth below, in connection with theappended drawings, is intended as a description of variousconfigurations and is not intended to represent the only configurationsin which the concepts described herein may be practiced. The detaileddescription includes specific details for the purpose of providing athorough understanding of the various concepts. However, it will beapparent to those skilled in the art that these concepts may bepracticed without these specific details. In some instances, well-knownstructures and components are shown in block diagram form in order toavoid obscuring such concepts.

Based on the teachings, one skilled in the art should appreciate thatthe scope of the disclosure is intended to cover any aspect of thedisclosure, whether implemented independently of or combined with anyother aspect of the disclosure. For example, an apparatus may beimplemented or a method may be practiced using any number of the aspectsset forth. In addition, the scope of the disclosure is intended to coversuch an apparatus or method practiced using other structure,functionality, or structure and functionality in addition to or otherthan the various aspects of the disclosure set forth. It should beunderstood that any aspect of the disclosure disclosed may be embodiedby one or more elements of a claim.

The word “exemplary” is used herein to mean “serving as an example,instance, or illustration.” Any aspect described herein as “exemplary”is not necessarily to be construed as preferred or advantageous overother aspects.

Although particular aspects are described herein, many variations andpermutations of these aspects fall within the scope of the disclosure.Although some benefits and advantages of the preferred aspects arementioned, the scope of the disclosure is not intended to be limited toparticular benefits, uses or objectives. Rather, aspects of thedisclosure are intended to be broadly applicable to differenttechnologies, system configurations, networks and protocols, some ofwhich are illustrated by way of example in the figures and in thefollowing description of the preferred aspects. The detailed descriptionand drawings are merely illustrative of the disclosure rather thanlimiting, the scope of the disclosure being defined by the appendedclaims and equivalents thereof.

Biological Impossibility or Implausibility in Conventional NeuralNetworks

Several common characteristics of conventional artificial neuralnetworks may be biologically impossible or implausible. Impossible orimplausible elements of conventional neural networks include non-neuronunits, weight inconsistency, non-local connectivity, overly informedlearning, and random initialization.

Non-neuron units may refer to putative single “neurons” that do notoutput a binary spike. This includes smoothed approximations of a stepfunction, such as a hyperbolic tangent (tanh) function, as well asfunctional forms derived from signal processing theory. A binary stepfunction may be a biologically possible model for a neuron under certainassumptions, as discussed below.

Weight inconsistency may refer to a property of conventional artificialneural networks by which synapses may change sign during training.Putative neurons may not have a uniform effect, either excitatory orinhibitory, on all downstream connections. According to Dale'sPrinciple, however, downstream synapses of a single neuron may beconstrained to have the same blend of neurotransmitters. For example,GABA and Glutamate may not mix within the same cell. Therefore, mixedexcitatory/inhibitory downstream synapses or sign changes on individualsynapses may not be biologically possible.

Non-local connectivity may refer to neural network models that containconnections between layers that do not reflect proximity constraints.For example, fully connected network layers may not reflect proximityconstraints. Biological neural connections may be constrained to acertain physical range. This physical range may correspond to localregions in the space of information being processed.

Overly informed learning may refer to neural network learning rules thatrely on mechanisms that may be biologically impossible or implausible.Such mechanisms may include backwards, direct, and/or real valuedinformation transfer between neurons and synapses.

Random initialization may refer to neural networks for which modelnetwork weights are initialized to a completely random state. As such,the network may not compute a meaningful response before training.Biological neural networks created in such a state may not support anorganism's ability to survive and learn. This form of biologicalimpossibility or implausibility may correspond to a low likelihood of acertain feature of biological neural networks that are subject tocompetitive evolutionary pressures.

Organic Learning

Aspects of the present disclosure avoid many or all of the biologicallyimpossible or implausible approaches or techniques described above. Inaddition, a neural network configured and/or trained in accordance withcertain aspects of the present disclosure may include one or morebiologically possible characteristics, as described below. As such, aneural network embodiment of certain aspects of the present disclosuremay be referred to as a biologically possible neural network. Systemsand methods in accordance with certain aspects of the present disclosuremay also be referred to as Organic Learning systems and methods.Biologically possible characteristics may enable a neural network tosolve a classification problem.

In one example, the neurons in a biologically possible neural networkmay make structured local patterns of connections between layers.Structured connections may imbue the network with built-in computationalfunctionality before training.

In a second example, a neural network may be configured with alternatinglayers. A first layer in the alternating pattern may include mixedpopulations of excitatory and inhibitory neurons. A second layer in thealternating pattern may include only neurons that are excitatory.

In a third example, neurons that are directly responsible for output mayfollow a learning rule based on their output. In some embodiments, thelearning rule may act upon the group of output neurons, as describedbelow. Interior (hidden layer) neurons may learn according to areinforcement rule. In some embodiments, interior (hidden layer) neuronsmay not learn at all.

In addition, a biologically possible neural network may be configuredsuch that mechanisms can be scaled down to simpler forms and stillfunction. In addition, mechanisms may be configured so that they may bescaled up to larger more complex networks. Scaling characteristics mayreflect the evolution of nervous systems from simpler to more complexforms, having useful properties at each evolutionary stage.

The following sections provide an example of how a biologically possibleneural network may be configured. Each layer of a neural network isdescribed from input to output, along with details of a specificimplementation concerning the classification of visual or auditory data.An exemplary neural network applied to visual data may classify digits.

Examples of digit visual data is illustrated in FIG. 1. The digits areblack and white anti-aliased digits on a 12×16 pixel frame. Poissonnoise is added to every presentation of the sample set. A first sample102 and a second sample 104 are illustrated. In one embodiment, no twopresentations or a digit are exactly the same.

Artificial Neurons

In one embodiment, artificial neurons in a biologically possible neuralnetwork may follow a simple step function. When the excitation (weightedsum of inputs) is below a configured threshold the output is zero. Whenthe excitation is above the threshold the output is either one ornegative one, depending on whether the neuron is excitatory orinhibitory.

In one embodiment, the threshold may be a fixed constant for allneurons, not an adjustable learning parameter. A fixed threshold maymore closely reflect the function of biological neurons. While firingthresholds of real neurons may be modulated, effects that modulate theresponse threshold of biological neurons may not reflect long termlearning. Because the threshold is fixed in the exemplary network, theweights of synapses may be described in units of the threshold. Thesedynamics may be summarized in Equation 1:

${e_{j} = {{\sum\limits_{i}^{\;}{w_{i}o_{i}\mspace{14mu} w_{i}}} > 0}},$

In Equation 1, “e” is the excitation given by the weighted sum of theinputs. The weight of the input from neuron “i” to neuron “j” is denoted“w_(i)”. Inputs to a neuron are the outputs, “o”, of neurons thatconnect to the neuron. The input from neuron “i” to neuron “j” isdenoted “o_(i)”. Inputs can be positive or negative. Weights areconstrained to be positive.

The output of a neuron can be summarized by Equation 2:o _(j) =s _(j) H(e _(j) −t) s _(i) Σ[−1,1]

The output, “o”, is given by the sign, “s”, of each neuron. The outputof neuron “j” is denoted “o_(j)”. The neuron fires whenever theexcitation is above a threshold, “t”, given by the Heaviside stepfunction “H”.

Artificial neurons that include time-dependent post-synaptic potentialsmay be referred to as “spiking” neurons. FIGS. 2A-B illustrate anembodiment of certain aspects of the present disclosure that includes asimulation of spiking neurons. As illustrated in FIG. 2A, a binaryoutput neuron may include a step function model of the Post-SynapticPotential (PSP). With a step function PSP, the precise time at which theinput spikes arrive may not matter as the receiving neuron, andtherefore the network, may reach the same final state irrespective of asmall jitter in time. In this example, three inputs arrived at theneuron, with the first contributing a first step 204 in the PSP. Thiswas followed by a second step 206 corresponding to the arrival of asecond input spike, and a third step 208 corresponding to a third inputspike. In this example, the three input spikes were similarly weighted.Collectively the three input spikes did not increase the PSP enough tocross the threshold 202. Since the fourth spike did not arrive beforethe steps down at the end of the PSP, this output neuron did not emit aspike in this time interval. Because the ordering of spikes may notmatter in a model of PSPs using a step function, a model may considerthat all inputs and outputs occur simultaneously.

If a PSP model that includes rise times and decay is used, as isillustrated in FIG. 2B, then the arrival timing and ordering of inputspikes may be more consequential. Still, even in this case, a singleepisode of PSP activations may correspond to that of a step-function PSPmodel in the case that the input neurons spike within a short time ofstimulus arrival. That is, if the input spikes arrive within a timewindow that is short relative to the decay of the PSP, the network mayreset before and after each episode of activity.

While the above description uses PSP functions to model spiking inputs,other forms of communication between neurons are also contemplated. Forexample, in some embodiments, a smoothed approximation using a tanh orsigmoid function may be used. In these cases, there may be no need foran explicit threshold, since the neurons may communicate theiractivation level to downstream neurons, rather than a binary outputbased on a threshold crossing.

According to Equation 1 and Equation 2, a neuron may receive positiveand/or negative inputs based on positive synaptic weights. The synapticweights may scale inputs from distinct excitatory and inhibitory units.That is, the scaling factor of each PSP (i.e. the weight) may beconstrained to be positive, but the input neurons themselves may beeither positive (excitatory) or negative (inhibitory). Other means forconfiguring a neural network to have positive and negative inputs to aneuron are also contemplated. In one example, the weights may haveeither positive or negative sign. That is, the weights may not beconstrained to be positive. In a second example, a neuron may have twooutputs, one which provides excitatory PSPs to downstream neurons andanother that provides inhibitory PSPs to downstream neurons. That is,the neurons may not be constrained to be either positive or negative,but may have different effects on different downstream neurons. In athird example, the first two examples may be intermixed. That is,neurons may have positive and negative outputs, and the weightsassociated with their connections to downstream neurons may also bepositive or negative (or zero).

In accordance with certain aspects of the present disclosure, a neuronmodel may include spiking or smoothed inputs that may be considered toarrive simultaneously. According to any of the above arrangements ofweight and input unit type constraints, the neurons in a subsequentlayer may be excited and/or inhibited by a pattern detected in a priorlayer.

Network Overview

An embodiment of certain aspects of the present disclosure may beapplied to classifying digits in visual data. One such embodiment mayinclude four feedforward layers. As illustrated in FIG. 3. The range ofoccupied positions may narrow through progressive layers, with Layer 1and Layer 2 neurons occupying X-positions in the range of positions 0-11(corresponding to the width of the input images) and Layer 3 and Layer 4neurons occupying X-positions in the narrower range of positions 2-9. Inthis example, the number of neurons per layer peaks in Layer 3.

According to certain aspects of the present disclosure, neurons may betopographically arranged, meaning that the position of every neuron maycorrespond to a location in an input space. For example, the position ofevery neuron may correspond to a pixel location if visual data is usedas input to the neural network. In addition, according to certainaspects of the present disclosure, connections between neurons may bemade based on this topography. For example, in some embodiments, whetheror not a neuron in Layer 3 connects to (receives input from) a neuron inLayer 2 may be based in part on the position of the Layer 3 neuron inthe topography and the position of the Layer 2 neuron in the topography.Additional examples of how connections between neurons may be based ontopography are provided below.

The network may include a population code for certain neuron types. Forexample, a set of neurons of a particular type may be repeated over thewhole topography. In this example, activation of a unit of thepopulation may indicate the relevance of its responsivity at thatposition.

The responsivity properties of a neuron may be created by specificpatterns of connections. Variations of these patterns may be repeatedwithout regard to whether they are likely to be activated in the digitsclassification task. For example, patterns may be rotated. In oneexample, each non-degenerate rotation of a pattern by a certain angle(e.g. 30 degrees) may be repeated as a separate population of neurons.Most neurons in this example may never fire during the presentation ofthe ten digits of the digit classification task. However, some of theneurons that do not fire during the digit classification task may fireduring other tasks. In this respect, a network that includessubstantially all non-degenerate variations of patterns may beconsidered to have a general network pattern that may be applied tosolve different problems.

Input Layer and Simple Features

FIGS. 4A-B illustrate how two different neurons may be configured tohave different selectivities in accordance with certain aspects of thepresent disclosure. These two different neurons may be considereddifferent types of Layer 2 neurons in a network configured to receive aone-dimensional input. In FIG. 4A and FIG. 4B, an input layer includesoverlapping sets of excitatory units 406 and inhibitory units 404. Fourunits (neurons) of each input type are shown. While the excitatory units406 and inhibitory units 404 are shown in staggered locations along thevertical axis, each pair of units may be considered to respond to thesame position in the one-dimensional topography.

In one configuration, the outputs of the excitatory neuron units 406 maybe real valued between 0 and 1. The value of the output may be inproportion to the inverted input intensity. If the input is an image, aninverted image intensity selectivity may correspond to high levels ofactivation for dark portions of the image and low values of activationfor light portions of the image. The input layer also containsinhibitory units 404 with real valued outputs between 0 and −1, ininverse proportion to the inverted image intensity. If the input is animage, an inverse of an inverted image intensity corresponds to highlevels of activation (a relatively large magnitude negative number) forlight portions of the image and low values of activation (a relativelysmall magnitude negative number) for dark portions of the image.

For the two-dimensional image classification network described above,there may be a two-dimensional array of excitatory neurons and a secondtwo-dimensional array of inhibitory neurons. Each array may beconsidered to occupy a plane in the topography. For the purpose ofconnectivity, the neurons from each plane may be considered to occupythe same Layer. In the example configuration described above, there maybe one excitatory neuron corresponding to each pixel in the input image.In addition, there may be one inhibitory neuron corresponding to eachpixel in the input image. These two neuron types are the two layer typesin the input layer (Layer 1) of the example digit classification neuralnetwork, as illustrated in the first row of FIG. 3.

As described above, the neurons in the input layers may output realvalued outputs between 0 and 1 (excitatory) or 0 and −1 (inhibitory).For the neurons in subsequent layers however, such as neuron 402 in FIG.4A and neuron 420 in FIG. 4B, the outputs may be configured to be binary(0 or 1). The use of real-valued outputs for an input layer may bedesirable for neuron layers that serve as sensory input neurons. Othermeans of sensory encoding are also contemplated. In one configuration, asensory variable at one location may be encoded by a population ofbinary spiking neurons. For example, each neuron in the population mayspike with a probability that is proportional to the value of thesensory variable. In an image processing task, each neuron in such apopulation could emit spikes with a probability that is proportional toimage intensity. In this way, the expected number of spikes emitted bypopulation of binary spiking neurons may closely match the value ofsensory variable.

In the digit classification example, the second layer contains neuronsthat are selective (i.e. receptive) to oriented edges or lines. Thislayer, therefore, may be referred to as having selectivity for “SimpleFeatures” as illustrated in FIG. 3. Selectivity for particular inputpatterns (such as edges or lines in visual data) may be considered aform of filtering. According to certain aspects of the presentdisclosure, the filtering function of neurons may result from configuredconnection patterns. In some embodiments, the filtering properties maybe present without learning. In addition, in some embodiments, afiltering property of a neuron that results from the configuration ofconnections may be further modified by learning.

FIG. 4A illustrates one embodiment of a neuron configured to have anedge detection filtering behavior. In this example, neuron 402, byvirtue of its connections to input layer neurons, will respond to anedge at a specific location in a one-dimensional topography. Given aninput layer with excitatory 406 and inhibitory 404 neurons correspondingto each position, the inhibitory neuron axon terminals 408 lie in oneplane and the excitatory axon terminals lie in another plane 410. Theposition of each axon terminal may be the same as the position of thecorresponding neuron body. In this example, the neuron 402 has twodendrites, a first dendrite 412 that is oriented upward from the cellbody 402, and a second dendrite 414 that is oriented downward from thecell body 414. The dendrites 412 and 414 extend through axon terminals408 and 410 of the inhibitory 404 and excitatory 406 input neurons,respectively. The connectivity pattern of the first dendrite 412 and thesecond dendrite 414 are not the same. Rather, the first dendrite 412makes connections (synapses), such as connection 416, with excitatoryaxon terminals in the upper region of the topography. In contrast, thesecond dendrite 414 makes connections (synapses), such as connection418, with inhibitory axon terminals in the lower region of thetopography. This configuration results in a selectivity for neuron 402that is responsive to an edge in the input. Specifically, the neuron 402may have a selectivity for an edge with a dark region above a lightregion at a particular location in the input topography corresponding tothe position of the neuron 402 in the topography. Likewise, otherneurons having the same connectivity pattern, but having differentpositions in the topography, may be selective for edges that appear inother regions of the topography.

FIG. 4B illustrates a second embodiment of a connectivity pattern inaccordance with certain aspects of the present disclosure. In thisexample, the neuron 420 may respond to a spot of intensity that isbounded by regions of low input intensity above and below. As with theexample illustrated in FIG. 4A, the example illustrated in FIG. 4Bincludes an input layer with excitatory 406 and inhibitory 404 neuronsresponding to each position in a one-dimensional topography. Theexcitatory neuron axon terminals 410 lie in one plane. The inhibitoryaxon terminals lie in a second plane 408. The neuron 420 has a centraldendrite 422 that makes synapses 424 with the axon terminals of theproximate excitatory neurons. The neuron 420 also has two sidedendrites, including an upper dendrite 426, that makes synapses 428 withthe axon terminals 408 of the inhibitory neurons 404. This configurationresults in a neuron that responds to a confined region of intensity inthe input that is bounded by low intensity regions. This selectivitypattern may be referred to an “on-center” and “off-periphery”.

While the connectivity patterns described with respect to FIG. 4A andFIG. 4B correspond to a one-dimensional topography, analogousconnectivity patterns may be applied to a two-dimensional topography.For example, a connectivity pattern analogous to FIG. 4B may beconfigured so that a neuron may be selective for a line in atwo-dimensional topography.

FIG. 5 illustrates examples of connection patterns for two-dimensionalinputs, as used for Layer 2 neurons in the example digit classificationnetwork. The inputs at this stage correspond to the input intensity ateach location in the topography. Accordingly, there is only a singleconnection made at each location to neurons in the second layer. Thefirst layer in this example is arranged so that there is one excitatoryand one inhibitory neuron at each location in the pixel topography. InLayer 2 and in subsequent layers, there may be multiple additional unittypes defined at each position.

As illustrated in FIG. 5, connections may be configured with a weighttemplate. In this example, each Layer 2 neuron may connect to inputs ina 4×4 pixel region of layer 1. Continuing with the general networkarchitecture that may be applied to digit classification, Layer 2 isconfigured to have eight neurons types, with each neuron type defined bya weight template. For example, the weight template 510 has excitatoryweights 518 in central positions and inhibitory weights 520 inperipheral positions on the left and right. The inputs from excitatoryweights 518 are illustrated in regular text on a white background. Theinputs from inhibitory neurons are illustrated with bold italic text ongray background. In each case, the sign of the text indicates the signof the neuron making a connection to the neuron at the correspondinglocation in the topography. A positive sign indicates that theconnection is with an excitatory neuron. A negative sign indicates thatthe connection is with an inhibitory neuron.

The connectivity patterns illustrated in FIG. 5 may be used to configurea neuron's selectivity for different two-dimensional patterns. Thesepatterns may correspond to the one-dimensional example illustrated inFIG. 4B. In each case, a neuron is configured by its weight template torespond when the center has a high intensity and the periphery has lowintensity. In a two-dimensional image this selectivity pattern maycorrespond to a line, and each template may correspond to a differentorientation of this line pattern. Weight template 502 corresponds to aselectivity for a substantially horizontal line. Weight template 504corresponds to a selectivity for a line that is tilted slightly upwardwith respect to horizontal. Weight template 506 corresponds to aselectivity for a line that is further tilted upward with respect tohorizontal. Weight template 508 corresponds to a selectivity for a linethat is tilted slightly rightward of vertical. Weight template 510corresponds to a selectivity for a substantially vertical line. Weighttemplate 512 corresponds to a selectivity for a line that is tiltedslightly leftward of vertical. Weight template 514 corresponds to aselectivity for a line that is tilted further leftward from vertical.Weight template 516 corresponds to a selectivity for a line that istilted still further leftward from vertical so that it is nearlyhorizontal. In the example network configuration, each location in layer2 of the network may be configured with all eight arrangements. Eacharrangement may be referred to as a neuron type.

In some embodiments, the weights may be configured with values thatdiffer from the values illustrated in FIG. 5A. For example, the weightassigned to each connection may be modified by randomly choosing a valuewithin a range of 80-100% of value shown in the template. In someembodiments, all of the weights for each neuron may be normalized to atotal weight that is around 6× the threshold, ‘t’, in Equation 2.

Complex Features

An embodiment of certain aspects of the present disclosure may includeneurons that respond to combinations of features. Continuing with thedigit classification example, Layer 3 neurons may respond to morecomplex features than the Layer 2 neurons just described. Theselectivity of Layer 3 neurons may be configured to respond tocombinations of the line segment features for which layer 2 neurons areselective. These combinations may yield selectivity for various curvesand shapes, as described below.

In accordance with certain aspects of the present disclosure, each Layer3 neuron may have a neuron body and one or more dendrites. The neuronbody may have a position in the same topography as the Layer 2 neurons.Each dendrite may be a curve in the topography starting from the neuronbody and extending into the topography. For example, the curve of adendrite may be a straight line. In the example of a straight line, thedendrite may be defined by an orientation and a length. Other curves arealso contemplated, including semi-circular curves, bended curves inmultiple directions, zig-zag patterns, and the like. In these examples,the dendrite may be defined by a length and a plurality of orientationsat different positions along the length.

Each dendrite of a neuron may extend into the topography where it maycome into proximity of one or more axon terminals of the Layer 2neurons. In the example digit classification network, the dendrites ofLayer 3 neurons project outwards from the position of the neuron body ina straight line. In addition, the length of the dendrite may beconfigured so that the length of the dendrite extends to the location ofthe closest axon terminal of layer 2 neurons. As described above, in theexample digit classification network, there may be axon terminals fromeach of eight Layer 2 neuron types at each pixel location in the visualdata topography.

Each dendrite may connect to a subset of the available input neurons.Continuing with the digit classification example, there may be eightavailable Layer 2 neurons for each dendrite of a layer 3 neurons. Theavailable Layer 2 neurons may have a range of selectivities. In thisexample, the available Layer 2 neurons will be selective for lineshaving one of the eight orientations described above.

According to certain aspects of the present disclosure, a processorcoupled to a memory may determine which subset of available neurons adendrite connects to based in part on the orientation of the dendrite atthat position. In the example network, the processor may determine foreach dendrite of the Layer 3 neurons whether to connect to a Layer 2.This determination may be based on the orientation of the dendrite inthe topography and the selectivities of the available neurons at thecorresponding position in the topography.

Different connection rules may yield neurons having a selectivity fordifferent complex features. Examples of different complex featureselectivities are provided in FIGS. 6A-C, 7A-C, 8A-B, 9A-B, and 10A-D.FIG. 11 summarizes the different types of complex feature detectors usedfor the aforementioned digit classification example.

FIGS. 6A-C illustrate a neuron configured to have a selectivity for acircle in accordance with certain aspects of the present disclosure. Inthis example, the oriented line detectors of layer 2 may be arranged incolumns. As described above, for one location in the topography, theremay be eight simple feature detectors in layer 2. Each of the eightneurons at the same location may be selective to a line at a differentorientation. For a portion of a Layer 3 dendrite that is proximate toone of these locations, therefore, there may be eight available Layer 2neurons/axon terminals with which it may connect. The collection of alleight available input neurons may be referred to as a column. In FIG.6A, a column 612 is located below and to the left of the layer 3 neuron602. For clarity, the column 612 is shown as being made up of sixdifferent line-detector neurons. In the example digit classificationnetwork, however, eight line-detector neurons were available at eachposition.

In FIG. 6A, the oriented line detectors are illustrated with shaded andunshaded regions corresponding to their pattern of excitatory andinhibitory connections. Filter arrangement 502, for example, maycorrespond to oriented line detector 614. Likewise, filter arrangement504 may correspond to oriented line detector 616. Filter arrangement 506may correspond to oriented line detector 618. Filter arrangement 510 maycorrespond to oriented line detector 620.

A neuron 602 that detects a complex feature is situated in the networkin the same topography as the layer 2 neurons 604 which respond to arange of oriented lines as described above. The neuron 602 has severaldendrites 606 that form synaptic connections 608 with Layer 2feature-detecting neurons 604. In this example, each dendrite forms aconnection with one or two of the available neurons in a proximatecolumn. For example, the dendrite that extends downward and leftwardfrom the cell body 602 forms a connection with a Layer 2 neuron that isselective to lines that point downward and rightward. The remainingdendrites likewise form connections with Layer 2 neurons that areselective to lines that point in a direction that is substantiallyperpendicular to the orientation of the dendrite at the location of theconnection.

In FIG. 6B, the connections that satisfy the above “substantiallyperpendicular” rule are illustrated at approximately equal distancesfrom the neuron body 602. As described above, each of the neurons in acolumn may be located at substantially the same position in thetopography. The radial offset of neurons in a column shown in FIG. 6Awas employed for clarity. By visualizing only the layer 2 neurons towhich the cell 602 forms connections, it can be appreciated that theneuron 602 will be selective to the presence of a circular shape 610.That is, the neuron 602 will respond to a circular shape when presentedin its receptive field.

The selectivity for a circular shape may be a consequence of configuringthe connections of the neuron based on the orientation of the dendriteat the location of each column of available neurons. Specifically, theconnections 608 may be configured so that connections are formed withoriented line detectors whose preferred orientation is within a range oforientations close to perpendicular to the dendrite orientation. Asthere may be no layer 2 line-detector neurons that have a preciselyperpendicular orientation to the dendrite, it may be desirable toconfigure connections that are within a range of the preciseperpendicular orientations.

Other embodiments are contemplated that may use other rules to determinewhether to connect to one or more available neurons. For example,whether a neuron connects to another neuron may be based on identifyingthe preferred selectivity of the neuron that most closely matches theorientation perpendicular to the orientation of the dendrite. In anotherexample, whether a neuron connects to another neuron may be based on theinput neuron having a preferred selectivity within a predetermined rangeof the orientation of the dendrite at that location. In the latterexample, a dendrite could make more than one connections at a givencolumn. In addition, a probability of a dendrite making a connectioncould be based on the orientation of a dendrite and the selectivity ofan input neuron. Alternatively, or in addition, a probability of adendrite making a connection may depend on a distance between thedendrite's corresponding neuron body and a position having availableneurons.

Continuing with the example of the digit classification network, thedistance at which connections are made may be closer in the verticaldirection of the topography and further in the horizontal direction inthe topography. As a result, the response may be strongest to anellipse, rather than a circle. While FIG. 6B illustrates an example ofselectivity to a circle, the results of the digit classificationnetwork, described below, used neurons having an elliptical selectivity.

Other possibilities are contemplated for which different rules areemployed to determine at what distance to make connections. In oneexample, a distance rule may specify a constant distance (to match acircle). In another example, a distance rule may depend on theorientation of the dendrite (to match a spiral). In still anotherexample, the distance rule may specify regular spaced intervals (tomatch concentric circles).

FIG. 6C illustrates a pattern in visual data corresponding to theselectivity of a complex feature detecting neuron as configured in theaforementioned digit classification network. The values illustrated inFIG. 6C reflect a linear superposition of the weights of the Layer 2neurons that form connections with complex feature detecting neuron. Thesign of the weights in FIG. 6C corresponds to whether the superpositionof weights affecting layer 2 neurons from the corresponding position ispredominantly excitatory (positive) or inhibitory (negative).

The illustration in FIG. 6C is not the same as the weight templatesillustrated in FIG. 5. In FIG. 5, the excitation of each Layer 2 neuronis the linear sum of the output of the Layer 1 neurons scaled accordingto the values of weight template. The excitation of the Layer 3 neuronillustrated in FIG. 6C, however, is not the linear sum of theactivations of Layer 2 neurons scaled by weights. Rather, the input tothe Layer 3 neuron corresponds to the output of the Layer 2 neurons,which may be binary according to a non-linear step function. The linearsuperposition of the weights of the Layer 2 neurons that formconnections to the Layer 3 neuron, therefore, may reflect the generalselectivity of the neuron, even though in any particular imagepresentation, some of the Layer 2 neurons may not emit a spike and maythus not contribute to the activation level of the Layer 3 neuron.

FIG. 6D illustrates a related embodiment of certain aspects of thepresent disclosure that is also used in the example digit classificationnetwork. In this example, a complex feature detector neuron in Layer 3may form connections with line detectors from Layer 2 so that the Layer3 neuron may be selective to half-ellipses. This example neuron has aset of dendrites 614 that span 180° around the neuron body 612. Thesedendrites 614 have synapse connections 608 that are made with orientedline detectors whose preferred orientation is approximatelyperpendicular to the dendrite direction as in FIG. 6A and FIG. 6B. As aresult of the connectivity rule and the relatively limited span ofdendrites, the neuron 612 responds to the presentation of a half circle616 in its receptive field. The arc to which the neuron is selective isin substantially the same position in the topography as the set ofdendrites 614 and the open side of the half ellipse is opposite theposition of the dendrites in the topography.

As with the line detectors of Layer 2 neurons described above, thecomplex feature detectors configured for the digit recognition examplemay also be characterized as having an orientation. For half-ellipsedetecting neurons, the open side of the half-ellipse to which differentneurons are selective may point in different directions in thetopography. In one example, a column of half circle detectors 618 maycontain a set of oriented half circle detectors that all respond to ahalf-ellipse at a single position in the topography, but with each unitresponding to a different orientation of half ellipse. One unit 620 of acolumn in FIG. 6E may respond to a half ellipse oriented in the samedirection as the neuron 612 illustrated in FIG. 6D. A second unit 622may respond to a half ellipse oriented at a 90° clockwise rotationrelative to orientation to which the neuron 612 is selective. A thirdunit 626 may respond to a half ellipse oriented at 180° relative to theorientation at which neuron 612 is selective. A fourth unit 624 mayrespond to a half ellipse presented at 270° relative to the orientationto which neuron 612 is selective. As illustrated in FIG. 6E, suchcolumns may be repeated over the topography in a similar manner as theoriented line detectors of the Simple Feature layer (Layer 2)illustrated in FIG. 6A.

As summarized in FIG. 11, two different types of semi-circle detectingneurons are used in the digit classification network example, with eachtype having a selectivity for a different size of semi-circle. Inaddition, neurons selective or sensitive to each size are included avariety of rotations and positions. Other embodiments of the presentdisclosure may include neurons that connect to line segment selectiveneurons spanning all or part of a circular, elliptical, ovoid or othercurved path at any orientation in the topography. As shown in FIG. 11,not all of the variations (e.g. rotations) of complex feature detectorneurons may actually fire in the context of the digits classificationexample. Still, the feature detectors may be deployed in a uniform wayas shown so that the network will may achieve a desired level ofperformance on a variety of classification problems.

FIGS. 7A-C illustrate another embodiment of certain aspects of thepresent disclosure that is applied in the digits classification example.In this embodiment, a neuron 702 is positioned in a topography proximateto Layer 2 neurons 704 that respond to a range of oriented lines asdescribed above. The oriented line detectors may be arranged in columns,such as the column 612 illustrated in FIG. 6A. The neuron 702 has twodendrites 706 that prefer to form synaptic connections 708 with Layer 2feature detecting neurons 704 that are selective to lines having asubstantially similar orientation to the dendrite's own orientation.This connectivity rule contrasts with the rule described in reference toFIGS. 6A-E in which the dendrites made connections to feature detectingneurons having a substantially perpendicular orientation to thedendrite's own orientation. In FIG. 7A, connections begin adjacent tothe neuron's location and continue along the length of the dendrites tothe edge of the neuron's receptive field. According to thisconfiguration of connections, the neuron 702 will respond to thepresence of a corner shape 710, as illustrated in FIG. 7B, whenpresented in its receptive field.

The connections in this example are formed with oriented line detectorshaving preferred orientations within a range around the dendriteorientation. FIG. 7C illustrates the linear superposition of the weightsof the Layer 2 neurons the connect to the neuron 702. The linearsuperposition therefore corresponds to the selectivity of the Layer 3neuron 702.

As shown in FIG. 11, other related neuron types used in theaforementioned digit classification example include those connectingline detectors to make complex features that respond to corners at avariety of angles, namely: 45 degrees, 70 degrees, 90 degrees, and 135degrees. Another related embodiment listed in FIG. 11 connects linedetectors with two dendrites at a 180 angle (parallel and on oppositesides of the neuron) to detect a long line. Another related embodimentmay use 3 or 4 dendrites to connect intersections of more than two linesat a point in the topography. As shown in FIG. 11, the digitsclassification example uses 3 and 4 line intersections at right angles.As in the case of the half-ellipse feature detectors illustrated in FIG.6D and FIG. 6E, the complex feature detectors described with referenceto FIGS. 7A-C may also be characterized as having an orientation.Likewise, columns corresponding to the same position and containingdifferent orientations of the complex feature may be arranged at eachpoint in the topography. The number of orientations included in theaforementioned digits classification network is shown in FIG. 11.

FIGS. 8A-B illustrate another embodiment of certain aspects of thepresent disclosure. This embodiment was not included in theaforementioned digit classification example but may be useful in othervisual classification problems. In this example, a neuron 802 issituated in the topography near to layer 2 neurons 804. The layer 2neurons 804 respond to a range of oriented edges (as illustrated forone-dimensional inputs with reference to FIG. 4A). The oriented edgesare arranged in columns. The neuron 802 has several dendrites 806 thatprefer to form synaptic connections 808 with feature detecting neurons804 that detect (i.e. are selective to) edges at a perpendicularorientation to the dendrite's own orientation at that position in thetopography. In this example, the connections are made at a shortdistance from the neuron 802 in all directions around the neuron body802. As a result of this pattern of connectivity, the neuron 802 willrespond to the presence of a solid circular shape 810, as illustrated inFIG. 8B, when the solid circular shape 810 is presented in the receptivefield of the neuron 802. Other embodiments of certain aspects of thepresent disclosure may be configured to be selective to a variety ofcontiguous solid shapes. The network arranged for the digitclassification example has no edge detectors like the one shown in FIG.8. Instead, the digit classification example includes line featuredetectors that are connected to line (rather than edge) selectiveneurons.

FIGS. 9A-B illustrates another embodiment of certain aspects of thepresent disclosure. As with the example shown in FIGS. 8A-B, the neurontype illustrated in FIGS. 9A-B was not included in the aforementioneddigit classification example, but may be useful in other visualclassification problems. A neuron 902 is situated in the topography nearto Layer 2 neurons 904 which respond to a range of oriented lines asdescribed above. The oriented line detectors may be arranged in columns,such as the column 612 as illustrated in FIG. 6A. The neuron 902 hasseveral dendrites 906 that prefer to form synaptic connections 908 withfeature detecting neurons that detect lines at a constant orientation,regardless of dendrite orientation. That is, the determination toconnect neuron 902 to neurons in the previous layer 904 may not be basedon the orientation of the dendrite, but instead may be based only on thepreferred selectivity of the putative input neurons. In this example,the synapse connections are made at regularly spaced distances alongeach of the dendrites. As a result of this connection pattern, theneuron 902 may respond to a texture that includes vertical lines 910.Other embodiments of certain aspects of the present disclosure maylikewise respond to other textures. Examples may include neuronsconfigured to be selective to patches of repeating patterns of multiplelines. The network created for the digit classification problem has noedge detectors like the one shown in FIGS. 9A-B.

Auditory Features

While the previous figures refer to an application to visualrecognition, certain aspects of the present disclosure may be applied tonon-visual data modalities, including other sensory modalities. FIGS.10A-D illustrate an embodiment of certain aspects of the presentdisclosure that may be applied to audio processing. In FIG. 10A, atopography of auditory inputs may be defined over a range of frequenciesmeasured in Hertz (Hz) and sound magnitudes measured in decibels (dB). Acollection of inhibitory neurons 1002 may be arranged in the topography.Likewise, a collection of excitatory neurons 1004 may be arranged in thetopography. Like the excitatory and inhibitory neurons in the visualexample, an instance of both an excitatory neuron 1002 and an inhibitoryneuron 1004 may be arranged at each position 1040 in the topography ofsound frequency and intensity. Other embodiments are also contemplatedin which the inhibitory and excitatory neurons may occupynon-overlapping regions of the auditory space topography.

In FIG. 10B, four filter arrangements are illustrated. A first filterarrangement 1006 responds when intensity is constant across a shortrange for frequencies. A second filter arrangement 1008 responds whenintensity increases across a range of frequencies. A third filterarrangement 1010 responds when intensity decreases across a range offrequencies. A fourth filter arrangement 1012 responds to only a narrowband of frequencies at a substantially constant intensity. These filterarrangements may be considered detectors of certain patterns offrequency and intensity in the auditory inputs. In the context of aneural network embodiment, the filter arrangements may be consideredinput neurons having a preferred selectivity.

FIG. 10C illustrates frequency/intensity input neurons that are tiledacross a broad range of frequencies and intensities in a columnarstructure. A neuron 1014 is in a layer of neurons receiving output fromthe frequency/intensity input neurons. In FIG. 10C, thefrequency/intensity neurons are illustrated with shaded and unshadedregions corresponding to their pattern of excitatory and inhibitoryconnections, respectively. Filter arrangement 1006, for example, maycorrespond to frequency/intensity neuron 1016. Likewise, filterarrangement 1008 may correspond to frequency/intensity filter 1018.Filter arrangement 1010, for example, may correspond tofrequency/intensity filter 1020. Filter arrangement 1012, for example,may correspond to frequency/intensity filter 1022.

As with the visual classification example, the filter/intensity filters(neurons) may be arranged in columns. The example illustrated in FIG.10C includes six columns. The bottom left column 1020 corresponds todifferent filter arrangements in a low frequency and low intensityregion of the topography.

The neuron 1014 has a first dendrite 1026 and a second dendrite 1028 towhich outputs from the frequency/intensity filters (neurons) may connect(form synapses), and thus affect the activation level of the neuron1014. In this example, the dendrite 1026 oriented in the direction oflower frequency and lower intensity from the neuron body 1014 connectsto filters 1030 and 1032 that have a preferred selectivity for lowerintensity and lower frequency within the range of frequencies andintensities covered by the topography. Likewise, dendrite 1028 orientedin the direction of higher frequency and lower intensity connects tofilters 1034 and 1036 that detect lower intensity at higher frequencieswithin the range of frequencies and intensities covered by thetopography.

FIG. 10D illustrates neuron 1014 superimposed over an auditory stimulus1038 for which the neuron 1014 may respond strongly. In this example,the auditory stimulus 1038 includes a broad range of frequencies. Theintensity is low at low frequencies, the increases with increasingfrequency, peaks, and then decreases with increasing frequency.

Output Layer

Continuing with the example of a digit classification network, an outputlayer may include neurons that receive inputs from Layer 3 neurons.Layer 3 neurons include all of the neuron types described above inreference to FIGS. 6A-E, 7A-B, and 11. In addition, in the digitclassification network, Layer 3 contains inhibitory versions of allthese neuron types. Both the excitatory and inhibitory version of aneuron may be configured from the same dendrite pattern, but withconnections to inputs that have an opposite valence.

As with the excitatory versions described above, the synaptic connectionweights may be configured to be within 80%-100% of the base valueaccording to a random modification. The total weight of synapses leadingto a neuron may then be normalized. For Layer 3 neurons, the weights maybe normalized to a value between 1.5× and 3× the neuron threshold. Thetotal synaptic weight may be determined so that the neuron respondsconsistently to the presentation of inputs that trigger its preferredselectivity in the presence of noise in the input. As illustrated inFIG. 1, each input image was presented to the network with added randomnoise.

When taking into consideration the inhibitory version of each neurontype, Layer 3 of the digit classification network may include a total ofaround 4,000 neurons.

FIG. 12 illustrates how an output layer (Layer 4) may include a regulardistribution of neurons based on the density of Layer 3 neurons. Eachoutput neuron takes all the excitatory input from a 2×2 pixel area 1202,1×2 pixel area 1204 or 1×1 pixel area 1206. Furthermore, FIG. 12 showsthe density of Layer 3 neurons at each position. For example, there are18 Layer 3 neurons at position 1208, which corresponds to (X=8,Y=13) inthe pixel topography. Layer 3 neurons are much denser in the centralarea. To offset the density in the central area, Layer 4 neurons in thecentral area have the smallest receptive fields, while Layer 4 neuronson the periphery have larger receptive fields.

The Layer 4 neurons are similar to Layer 2 neurons in that they receiveboth excitatory and inhibitory inputs and have a high normalizedsynaptic weight compared to the normalized synaptic weight of Layer 3neurons, which only receive excitatory inputs. Unlike Layer 2 neurons,however, the Layer 4 neurons are initialized to connect to excitatoryand inhibitory inputs within their receptive field in an initiallyrandom pattern.

For the example digit classification network, the pattern of connectionsfor Layer 4 neurons are configured according to the following method.First, receptive fields are determined based on the density of layer 3neurons. As the receptive field corresponds to the length of thedendrite, this first step may also be considered determining a length ofLayer 4 neuron dendrites based on the density of Layer 3 neuronsproximate to the Layer 4 neurons in the topography. Second, the Layer 4neurons connect to all of the available neurons that are proximate totheir dendrites. The connections are initialized with synaptic weightsthat are randomly set to +/−20% of a base value. Third, the weights arenormalized so that the magnitude of the excitatory weights are threetimes the magnitude of the inhibitory weights and so that the totalweight is five times the threshold, t.

For a multi-class classification task, such as the digit classificationtask, this connectivity method may be repeated for each category. Sincethere are ten digits, this process may be repeated ten times, once foreach of the digits between 0 and 9. Accordingly, each output categorymay be configured with its own complete set of connections over Layer 3.Since the weights to the Layer 3 neurons are initially chosen at random,the weights may then be adjusted by the learning algorithm. Theclassification given by the network may be determined by selecting theoutput group that has the highest firing rate in response to the input.

The learning rule may also apply to bias units. As described above, thethreshold for each neuron may be configured with a constant value.Still, each neuron may also receive excitatory and inhibitory biasinputs that may be adjusted according with learning. The bias neuronsmay be neurons that fire in response to any input, regardless of whatinput is presented to the network. The excitatory and inhibitory biasneurons may be initialized with equal weights so they may havesubstantially no effect prior to training. The associated weights maythen be adjusted as part of the network training. Alternatively, or inaddition, the neurons may be configured so that the threshold of eachneuron is an adjustable parameter. It may be desirable to configureneurons with fixed thresholds, however, as this may facilitatecomparisons with biologically plausible learning mechanisms. In theexample digit classification network, bias units are only applied to theoutput layer neurons. Furthermore, in the example network, the interiorlayers are untrained. Other configurations are also contemplated. Forexample, network configurations in which bias neurons are also used forinterior layers is also contemplated.

Training Algorithm

In accordance with certain aspects of the present disclosure, a trainingalgorithm may be applied to modify weights leading to output layerneurons. In one example, a learning rule may apply a global supervisionsignal in combination with local information at each synapse. Thisexample may be considered similar to the Perceptron learning algorithm.The learning rule may be applied to each group of neurons correspondingto an output category separately. That is, each group of neurons may betrained on a one-against all classification of their preferred target.Given a learning rate parameter l and a number n of targets, thealgorithm applied to each group after each example presentation mayinclude the following steps: First, if the example is the target, thetraining signal is l. If, instead, the example is a non-target, thetraining signal is given as:

$- {\frac{l}{n - 1}.}$

Second, for each synaptic weight, the weight update is determined basedon the training signal multiplied by the input value on the synapse.That is: w→w+ir.

The scaling of the training signal is uneven for target and non-targetpresentations because the non-target presentations are naturally morenumerous by a ratio of (n−1) to 1. This scaling factor difference,however, may be considered an optional design choice. In addition,learning may be applied to interior layers. However, the results in thenext section are based upon simulations in which the learning rule wasonly applied to output layer neurons.

Test Results

Test results are presented before and after training. Test results froma neural network prior to training may be referred to as untrainedperformance.

Before training, the model output was compared on the set of targetdigits in comparison to a set of non-digit distractors. The distractorsare shown in FIG. 13. The distractors include a dark image with noise1302, a solid circle 1304, a solid rectangle 1306, a solid star 1308, ascattering of small boxes 1310, white noise 1312, a group of ellipsesarranged as a face 1314, a group of rectangles 1316, a diffuse circle1318, and a checkerboard pattern 1320. Each distractor is presented withadditive noise, the same as for the target digits shown in FIG. 1.

The number of output neurons firing in response to each digit andnon-digit is shown in FIG. 14. The digits are listed by the digit numberitself, while the distractors are listed by reference number in FIG. 13.Alongside each digit and distractor is the average number of outputlayer neurons that responded to a presentation of that image in eachoutput group. Since there were 68 output neurons per group, the maximumvalue in this test is 68. As can be appreciated with reference to FIG.14, the network prefers the digits to non-digits in every case. That is,every digit has a firing rate higher than every non-digit. Overall, thenetwork achieves what could be described as perfect classification of onthis digit vs. distractor test.

FIG. 15 illustrates how the network performance for digit classificationimproves after each epoch of training. An epoch may refer to thepresentation of each target image once. The target images may bepresented in a random order. The average accuracy 1502 is shown with athick black line, while the progress of several example training runs1504 are shown in thin gray lines. Although the network has an initialpreference of digits over non-digit inputs, the untrained network has nopreference for any particular digit. Before training, networkperformance on the digit classification task (as opposed to the digitvs. distractor task) is completely random. The average accuracy is 10%on average, prior to training as illustrated in FIG. 15. This level ofperformance is expected for a classification task involving 10categories.

As described above, the output category may be determined as the groupwith the highest firing rate (out of 68 output neurons in each group).While all networks achieve perfect accuracy at some point in theirtraining, the noise in the sample perturbs some trained networks and thehighest overall (average) accuracy achieved at any point is 98%. Thislevel of accuracy is achieved after around 7 or 8 presentations of eachimage. As shown in FIG. 15, some networks may achieve a highclassification rate after just one or two presentations of each image.

The high level of accuracy in a short training time may reflect theutility of certain aspects of the present disclosure. In particular, byconfiguring neurons with specific patterns of connectivity, theconfigured structures may obviate most learning in the network. In theexample object classification network, only the output layer had to betrained. The response properties of the interior neurons had apparentlyalready transformed the inputs well enough so that weight modificationson the output later alone could yield satisfactory performance.

In addition, even with no training, the preconfigured patterns ofconnectivity were shown to be capable of distinguishing digits fromnon-digit distractors.

Furthermore, in comparison to some current machine learning techniques,the pre-configuration of connection patterns may result in a more sparseconnectivity for the network as a whole, which in turn may be amenableto desirable computational properties of sparse matrices.

As used herein, the term “determining” encompasses a wide variety ofactions. For example, “determining” may include calculating, computing,processing, deriving, investigating, looking up (e.g., looking up in atable, a database or another data structure), ascertaining and the like.Additionally, “determining” may include receiving (e.g., receivinginformation), accessing (e.g., accessing data in a memory) and the like.Furthermore, “determining” may include resolving, selecting, choosing,establishing and the like.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: a, b, or c” is intended to cover: a, b, c,a-b, a-c, b-c, and a-b-c.

The methods disclosed herein comprise one or more steps or actions forachieving the described method. The method steps and/or actions may beinterchanged with one another without departing from the scope of theclaims. In other words, unless a specific order of steps or actions isspecified, the order and/or use of specific steps and/or actions may bemodified without departing from the scope of the claims.

The processing system may be configured as a general-purpose processingsystem with one or more microprocessors providing the processorfunctionality and external memory providing at least a portion of themachine-readable media, all linked together with other supportingcircuitry through an external bus architecture. Alternatively, theprocessing system may comprise one or more specialized processors forimplementing the neural networks, for example, as well as for otherprocessing systems described herein.

Thus, certain aspects may comprise a computer program product forperforming the operations presented herein. For example, such a computerprogram product may comprise a computer-readable medium havinginstructions stored (and/or encoded) thereon, the instructions beingexecutable by one or more processors to perform the operations describedherein. For certain aspects, the computer program product may includepackaging material.

Further, it should be appreciated that modules and/or other appropriatemeans for performing the methods and techniques described herein can bedownloaded and/or otherwise obtained by a user terminal and/or basestation as applicable. For example, such a device can be coupled to aserver to facilitate the transfer of means for performing the methodsdescribed herein. Alternatively, various methods described herein can beprovided via storage means (e.g., RAM, ROM, a physical storage mediumsuch as a compact disc (CD) or floppy disk, etc.), such that a userterminal and/or base station can obtain the various methods uponcoupling or providing the storage means to the device. Moreover, anyother suitable technique for providing the methods and techniquesdescribed herein to a device can be utilized.

It is to be understood that the claims are not limited to the preciseconfiguration and components illustrated above. Various modifications,changes and variations may be made in the arrangement, operation anddetails of the methods and apparatus described above without departingfrom the scope of the claims.

What is claimed is:
 1. A system for configuring an artificial neuralnetwork, comprising: a memory; and a processor coupled to the memory,wherein the processor is configured to: arrange a first plurality ofneurons in a topography, wherein a selectivity of a first neuron of thefirst plurality is based at least in part on a position of the firstneuron in the topography; arrange a second plurality of neurons in thetopography; wherein a second neuron of the plurality of neurons includesa body and a dendrite, the body having a position in the topography andthe dendrite extending from the body and having an orientation in thetopography, wherein a number of neurons in the second plurality ofneurons is based on a number of non-degenerate variations of aconnectivity pattern, wherein a variation of the connectivity pattern isrotated from a second variation of the connectivity pattern by apredetermined angle; and connect the first neuron to the second neuron,so that an output of the first neuron is an input to the second neuron,based at least in part on: the position of the first neuron; theorientation of the dendrite at the position of the first neuron; and theselectivity of the first neuron.
 2. The system of claim 1, wherein theprocessor is configured to connect the first neuron to the second neuronfurther based on: the position of the body of the second neuron.
 3. Thesystem of claim 1, wherein a number of connections to the second neuronis based on a predetermined range around a target number of connectionsfor the second plurality of neurons.
 4. The system of claim 1, whereinthe processor is configured to connect the first neuron to the secondneuron further based on: a density of available neurons at the positionof the first neuron.
 5. The system of claim 1, wherein a length of thedendrite depends on a density of neurons in the first plurality ofneurons at positions proximate to the position of the body of the secondneuron.
 6. The system of claim 1, wherein the processor is furtherconfigured to connect a dendrite of a downstream neuron to one or moreneurons of the second plurality based at least in part on: a preferredconnectivity pattern; and the predetermined angle between non-degeneratevariations of the connectivity pattern.
 7. The system of claim 1,wherein the processor is further configured to: arrange a thirdplurality of neurons in the topography, wherein each connection fromneurons of the third plurality to the second neuron has a negativevalence, and wherein each connection between neurons in the firstplurality and the second neuron has a positive valence.
 8. The system ofclaim 1, wherein the processor is further configured to: receive visualdata; and classify the visual data based on a neural network, whereinthe neural network comprises the first plurality of neurons and thesecond plurality of neurons.
 9. The system of claim 1, wherein theprocessor is further configured to: receive auditory data; and classifythe auditory data based on a neural network, wherein the neural networkcomprises the first plurality of neurons and the second plurality ofneurons.
 10. A non-transitory computer readable medium havinginstructions stored thereon that, upon execution by a computing device,cause the computing device to perform operations comprising: arranging afirst plurality of neurons in a topography, wherein a selectivity of afirst neuron of the first plurality is based at least in part on aposition of the first neuron in the topography; arranging a secondplurality of neurons in the topography; wherein a second neuron of theplurality of neurons includes a body and a dendrite, the body having aposition in the topography and the dendrite extending from the body andhaving an orientation in the topography; connecting the first neuron tothe second neuron, so that an output of the first neuron is an input tothe second neuron, based at least in part on: the position of the firstneuron; the orientation of the dendrite at the position of the firstneuron; and the selectivity of the first neuron; and arrange a thirdplurality of neurons in the topography, wherein each connection fromneurons of the third plurality to the second neuron has a negativevalence, and wherein each connection between neurons in the firstplurality and the second neuron has a positive valence.
 11. Thenon-transitory computer readable medium of claim 10, wherein connectingthe first neuron to the second neuron is further based on: the positionof the body of the second neuron.
 12. The non-transitory computerreadable medium of claim 10, wherein a number of connections to thesecond neuron is based on a predetermined range around a target numberof connections for the second plurality of neurons.
 13. Thenon-transitory computer readable medium of claim 10, wherein connectingthe first neuron to the second neuron is further based on: a density ofavailable neurons at the position of the first neuron.
 14. Thenon-transitory computer readable medium of claim 10, wherein a length ofthe dendrite depends on a density of neurons in the first plurality ofneurons at positions proximate to the position of the body of the secondneuron.
 15. The non-transitory computer readable medium of claim 10,wherein a number of neurons in the second plurality of neurons is basedon a number of non-degenerate variations of a connectivity pattern,wherein a variation of the connectivity pattern is rotated from a secondvariation of the connectivity pattern by a predetermined angle.
 16. Thenon-transitory computer readable medium of claim 15, having instructionsstored thereon that, upon execution by a computing device, cause thecomputing device to perform operations further comprising: connecting adendrite of a downstream neuron to one or more neurons of the secondplurality based at least in part on: a preferred connectivity pattern;and the predetermined angle between non-degenerate variations of theconnectivity pattern.
 17. The non-transitory computer readable medium ofclaim 10, having instructions stored thereon that, upon execution by thecomputing device, cause the computing device to perform operationsfurther comprising: receiving visual data; and classifying the visualdata based on a neural network, wherein the neural network comprises thefirst plurality of neurons and the second plurality of neurons.
 18. Amethod of configuring an artificial neural network, comprising:arranging a first plurality of neurons in a topography, wherein aselectivity of a first neuron of the first plurality is based at leastin part on a position of the first neuron in the topography; arranging asecond plurality of neurons in the topography; wherein a second neuronof the plurality of neurons includes a body and a dendrite, the bodyhaving a position in the topography and the dendrite extending from thebody and having an orientation in the topography; and connecting thefirst neuron to the second neuron, wherein a number of connections tothe second neuron is based on a predetermined range around a targetnumber of connections for the second plurality of neurons, so that anoutput of the first neuron is an input to the second neuron, based atleast in part on: the position of the first neuron; the orientation ofthe dendrite at the position of the first neuron; and the selectivity ofthe first neuron.
 19. The method of claim 18, wherein connecting thefirst neuron to the second neuron is further based on: the position ofthe body of the second neuron.
 20. The method of claim 18, wherein anumber of neurons in the second plurality of neurons is based on anumber of non-degenerate variations of a connectivity pattern, wherein avariation of the connectivity pattern is rotated from a second variationof the connectivity pattern by a predetermined angle.